class: center, middle, inverse, title-slide .title[ # ISA 419: Data-Driven Security ] .subtitle[ ## 16: Machine Learning Regression Models ] .author[ ### <br>Fadel M. Megahed, PhD <br><br> Professor <br> Farmer School of Business<br> Miami University<br><br> <a href="https://twitter.com/FadelMegahed"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z" /></svg> <span class="citation">@FadelMegahed</span></a> <br> <a href="https://github.com/fmegahed/"><svg viewBox="0 0 496 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z" /></svg> fmegahed</a> <br> <a href="mailto:fmegahed@miamioh.edu"><svg viewBox="0 0 512 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M476 3.2L12.5 270.6c-18.1 10.4-15.8 35.6 2.2 43.2L121 358.4l287.3-253.2c5.5-4.9 13.3 2.6 8.6 8.3L176 407v80.5c0 23.6 28.5 32.9 42.5 15.8L282 426l124.6 52.2c14.2 6 30.4-2.9 33-18.2l72-432C515 7.8 493.3-6.8 476 3.2z" /></svg> fmegahed@miamioh.edu</a><br> <a href="https://calendly.com/fmegahed"><svg viewBox="0 0 384 512" style="height:1em;position:relative;display:inline-block;top:.1em;fill:white;" xmlns="http://www.w3.org/2000/svg"> <path d="M202.021 0C122.202 0 70.503 32.703 29.914 91.026c-7.363 10.58-5.093 25.086 5.178 32.874l43.138 32.709c10.373 7.865 25.132 6.026 33.253-4.148 25.049-31.381 43.63-49.449 82.757-49.449 30.764 0 68.816 19.799 68.816 49.631 0 22.552-18.617 34.134-48.993 51.164-35.423 19.86-82.299 44.576-82.299 106.405V320c0 13.255 10.745 24 24 24h72.471c13.255 0 24-10.745 24-24v-5.773c0-42.86 125.268-44.645 125.268-160.627C377.504 66.256 286.902 0 202.021 0zM192 373.459c-38.196 0-69.271 31.075-69.271 69.271 0 38.195 31.075 69.27 69.271 69.27s69.271-31.075 69.271-69.271-31.075-69.27-69.271-69.27z" /></svg> Automated Scheduler for Office Hours</a><br><br> ] .date[ ### Spring 2025 ] --- ## Quick Refresher of Last Class ✅ Describe the basic concepts of regression analysis, including the roles of independent and dependent variables. ✅ Assess regression models using metrics like R-squared and Mean Squared Error and interpret the results. ✅ Describe the two modeling mindsets: explanatory and predictive. --- ## Learning Objectives for Today's Class - Recognize the differences and similarities between traditional regression and machine learning regression models - Describe advanced techniques like Ridge and Lasso - Explore non-linear models like Decision Trees and Random Forests for regression tasks - Apply machine learning regression models to cyber security data sets --- class: inverse, center, middle # Differences Between Traditional (Statistical) and Algorithmic (Machine Learning) Modeling Cultures --- ## Statistical Modeling Culture Find a stochastic model of the data-generating process in the form of: `\(Y = f(X, \, parameters, \, \epsilon)\)` <img src="data:image/png;base64,#../../figures/stat_model_culture.png" width="25%" style="display: block; margin: auto;" /> .pull-left[ .center[**Assumptions**] .font90[ - Specific stochastic model of the data-generating process - Distributional assumptions about the errors - Linearities - Manual specification of interactions ] ] .pull-right[ .center[**Problems**] .font90[ - Conclusions about model, and not about "nature" - Assumptions often violated - Often no model evaluation (no focus on prediction) ] ] .footnote[ <html> <hr> </html> **Image Source**: Chris Molnar. (2014). [Statistical Modeling: The Two Cultures - Based on Leo Breiman's paper](https://www.slideshare.net/christophmolnar/presentation-on-statistical-modeling-the-two-cultures) ] --- ## Algorithmic (Machine Learning) Modeling Culture Find a function `\(f(x)\)` that minimizes the loss `\(L(Y, \, f(X))\)` <img src="data:image/png;base64,#../../figures/alg_model_culture.png" width="25%" style="display: block; margin: auto;" /> .pull-left[ .center[**Assumptions**] .font90[ - No assumptions about the data-generating process (true mechanism of generating the data is **unknown** and not of direct interest). - No assumptions about the distribution of the errors. ] ] .pull-right[ .center[**Problems**] .font90[ - Some models are "black boxes". - Stability of the model is not guaranteed (and honestly this is also a problem in statistical modeling). ] ] .footnote[ <html> <hr> </html> **Image Source**: Chris Molnar. (2014). [Statistical Modeling: The Two Cultures - Based on Leo Breiman's paper](https://www.slideshare.net/christophmolnar/presentation-on-statistical-modeling-the-two-cultures) ] --- ## Statistical Modeling: The Two Cultures <img src="data:image/png;base64,#../../figures/two_cultures.png" width="40%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Note**: You are highly encouraged to read Leo Breiman's paper on the two cultures of statistical modeling. It's a classic and a must-read for anyone interested in data science and machine learning. See [Breiman 2001](https://www.jstor.org/stable/2676681). ] --- class: inverse, center, middle # Advanced Techniques for Regression --- ## Ridge Regression <img src="data:image/png;base64,#../../figures/Ridge_Regression_print.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chris Albon. (2017). [Machine Learning Flashcards](https://machinelearningflashcards.com/). Flashcards were purchased by the author and are not available for free. ] --- ## Lasso Regression <img src="data:image/png;base64,#../../figures/Lasso_For_Feature_Selection_print.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chris Albon. (2017). [Machine Learning Flashcards](https://machinelearningflashcards.com/). Flashcards were purchased by the author and are not available for free. ] --- class: inverse, center, middle # Non-Linear Models for Regression --- ## Decision Trees (Classification and Regression Trees) <img src="data:image/png;base64,#../../figures/Decision_Trees_print.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chris Albon. (2017). [Machine Learning Flashcards](https://machinelearningflashcards.com/). Flashcards were purchased by the author and are not available for free. ] --- ## Decision Trees (Classification and Regression Trees) <img src="data:image/png;base64,#../../figures/Decision_Tree_Regression_print.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chris Albon. (2017). [Machine Learning Flashcards](https://machinelearningflashcards.com/). Flashcards were purchased by the author and are not available for free. ] --- ## Random Forests <img src="https://miro.medium.com/v2/resize:fit:4800/format:webp/1*ZFuMI_HrI3jt2Wlay73IUQ.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chaya. (2020). [Random Forest Regression](https://levelup.gitconnected.com/random-forest-regression-209c0f354c84). ] --- ## Random Forests <img src="data:image/png;base64,#../../figures/The_Random_In_Random_Forest_print.png" width="72%" style="display: block; margin: auto;" /> .footnote[ <html> <hr> </html> **Image Source:** Chris Albon. (2017). [Machine Learning Flashcards](https://machinelearningflashcards.com/). Flashcards were purchased by the author and are not available for free. ] --- class: inverse, center, middle # Applications to Cybersecurity Datasets --- ## Example: Predicting Flow Duration in IoT Networks ``` python from ucimlrepo import fetch_ucirepo # fetch dataset rt_iot2022 = fetch_ucirepo(id=942) # data (as pandas dataframes) X = rt_iot2022.data.features y = rt_iot2022.data.targets df = pd.concat([X, y], axis=1) # subset the features and dataset df = ( df .sample(frac=0.1, random_state=2025) .reset_index(drop=True) .loc[:, ['proto', 'service', 'fwd_pkts_payload.avg', 'bwd_pkts_payload.avg', 'fwd_subflow_pkts', 'bwd_subflow_pkts', 'Attack_type', "flow_duration"]] ) ``` .footnote[ <html> <hr> </html> **Note**: The dataset is from the UCI Machine Learning Repository. It contains features of network traffic in IoT networks and the target variable is the flow duration. The goal is to predict the flow duration using the features. For more information, see [UCI RT-IoT2022](https://archive.ics.uci.edu/dataset/942/rt-iot2022). ] --- class: inverse, center, middle # Recap --- ## Summary of Main Points By now, you should be able to do the following: - Recognize the differences and similarities between traditional regression and machine learning regression models - Describe advanced techniques like Ridge and Lasso - Explore non-linear models like Decision Trees and Random Forests for regression tasks - Apply machine learning regression models to cyber security data sets --- ## 📝 Review and Clarification 📝 1. **Class Notes**: Take some time to revisit your class notes for key insights and concepts. 2. **Zoom Recording**: The recording of today's class will be made available on Canvas approximately 3-4 hours after the end of class. 3. **Questions**: Please don't hesitate to ask for clarification on any topics discussed in class. It's crucial not to let questions accumulate.